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Abstract 

Open Educational Resources (OER) are gaining acceptance as legitimate and effective teaching materials 
in higher education, particularly in 2-year institutions. Despite the steady growth in the availability and 
use of OER, there have been relatively few studies on the efficacy of OER and student achievement. This 
study analyzed the effect enrollment in an OER course had on student academic achievement when 
controlling for prior academic achievement in an introductory online history course at a large community 
college. Correlation analysis and simple linear regression were conducted. The results of this research 
indicate a significant positive moderate correlation between OER and student achievement. The study 
provided a process by which future, more rigorous efficacy studies can be conducted. 
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Introduction 

For the past decade, a more open approach to learning has emerged. The Open Educational Resources, or 
OER, movement is changing the way students learn and institutions operate. The most widely used 
definition of OER comes from the William and Flora Hewlett Foundation: 

OER are teaching, learning, and research resources that reside in the public domain or have been 
released under an intellectual property license that permits their free use and re-purposing by 
others. Open educational resources include full courses, course materials, modules, textbooks, 
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streaming videos, tests, software, and any other tools materials or techniques used to support 
access to knowledge. (2013, para. 2) 

The Open Movement began in a wider context of a culture of open knowledge and peer collaboration. 
More recently, the OER movement has gained traction, particularly in American community colleges with 
their open admissions policies and inclusive philosophies. OER have proven to be a key component to 
student access and success, providing free or low-cost textbook alternatives to financially strapped 
students (Bradley, 2013). As more community colleges are adopting OER (Ashford, 2017), it is important 
to examine the impact of OER beyond cost saving to students. 


Literature Review 

To date, there have been relatively few studies on the efficacy of OER and student achievement. Robinson, 
Fischer, Wiley, and Hilton III (2014) used a quantitative quasi-experimental design to determine whether 
the choice of open textbooks had a significant impact on student science learning in public high school. 
After controlling for teacher efficacy and student characteristics, the researchers found that the use of 
OER did have a statistically significant effect on student learning in chemistry but not in biology or earth 
systems as demonstrated by student scores on end-of-year state standardized test scores. This study 
demonstrated that the adoption of OER can both dramatically reduce cost for students while at the same 
time improve student learning or, at the minimum, not negatively impact student learning. This study 
took place in a secondary setting, differentiating it from other studies reviewed here. Even so, the study 
does provide a framework for further efficacy studies, whether in secondaiy or post-secondaiy school 
settings. Robinson (2015) examined Open Educational Resource (OER) adoption as a correlate of post¬ 
secondary school students’ academic success. Using an ex-post facto quasi-experimental research design, 
the study analyzed the academic performances of students using the open textbooks and those that were 
using the hardcopy traditional textbooks in seven Project Kaleidoscope post-secondary educational 
institutions. Having controlled for selection bias using propensity score matching (PSM), multiple 
regression and logistic regression models were used to analyze the effect of the OER on the students’ 
academic performance. The findings showed that students using the traditional textbooks performed 
relatively better than those using the open textbooks in business and psychology courses. At the same 
time, students in this study who used open textbooks enrolled in more credits than students who used 
textbooks. 

Fischer, Hilton III, Robinson, and Wiley (2015) analyzed the 2 nd year of OER adoption in the same 
schools as did Robinson (2015) (who examined the first year of adoption). They sought to know whether 
the adoption of no-cost digital textbooks had an impact on post-secondaiy students’ learning outcomes. 
The study examined course completion, class achievement, and intensity of enrollment during and after 
semesters in which students used OER and compared these to the same outcomes for students enrolled in 
the same courses that did not use OER. The sample was large, including 16,727 students, 4909 who were 
in the treatment group and 11,818 who were in the control group enrolled in 15 different undergraduate 
classes across 10 different institutions spread out across the United States. The size of this study makes 
the results more generalizable to a larger population than any other study of its kind conducted to date. 


232 



The Impact of Enrollment in an OER Course on Student Learning Outcomes 
Grewe and Davis 


There were statistically significant differences in enrollment intensity between students enrolled in 
courses which used OER and those that did not, with students who used OER taking more credits than 
their counterparts. There were also moderate differences in completion rates and final grades between the 
two groups, with those in OER courses performing as well as or better than their counterparts in courses 
where faculty used commercial textbooks. 

Hilton III, Gaudet, Clark, Robinson and Wiley (2013) examined the implications of the adoption of OER 
in a Math Department of the Scottsdale Community College (SCC) in Arizona. The large sample size 
included 2,043 students. Specifically, the study examined the cost benefits of OER and the impact on the 
students’ retention as well as the perception of the students and the faculty about the quality of the OER. 
The study was conducted in Fall 2012. The metrics used in measuring the students’ learning outcomes 
include the withdrawal rate and C grade score or better reported by the college. Findings revealed that use 
of the open textbooks resulted in a significant cost saving. However, there was no significant change in the 
withdrawal rate and the students' retention level attributable to the use of the open textbooks. In terms of 
perception of the open textbooks' quality, both the students and the faculty perceived the books as being 
of good quality. 

Hilton III and Laman (2012) investigated the efficacy of online open textbooks in improving students’ 
academic performance at Houston Community College (HCC). The pilot study assessed the academic 
performance of students who used the traditional texts in the Spring 2011 semester and those who used 
open textbooks during the Fall 2011 semester. Grade point average, withdrawal rate, and departmental 
examination scores constituted the performance measures. Findings revealed that students who used the 
free online textbook had better overall outcomes than students who used the traditional textbooks. The 
study, however, could not establish a causal relationship. Future studies are needed to establish the use 
and adoption of OER as the driver of positive student learning outcomes. 

Feldstein et al. (2012) also examined the impact of the adoption of the open textbooks on access and 
learning outcomes of students in the Virginia State University School of Business in a pilot study. All the 
participating students took courses that used Flat World Knowledge (FWK) open online textbooks 
between the Fall 2010 and Spring 2011. Students were also provided access to an array of supplementary 
online resources. Grades in the courses were used as measures of the students’ learning outcomes. The 
researchers compared the grades of students in core courses using the FWK open text and materials to 
students enrolled in the same core courses that used traditional texts. The results revealed that there was 
a significant (p<o.oi) difference in the students’ performance in courses taught with the open online 
textbooks, having controlled for students’ previous academic performance. However, this study has the 
important caveat that the courses were not identical and the difference in courses could have dwarfed any 
impact of open textbooks. 

Fovett, Meyer, and Thille (2008) evaluated the effectiveness of the hybrid Open Teaming Initiative (OLI) 
Statistics course developed at Carnegie Mellon University (CMU) in what they called an “Accelerated 
Teaming” study in the Spring of 2007. The study employed a quasi-experimental study design in which 
students were assigned to different groups: those who used OFI-Statistics to supplement classroom 
instruction or those who received the traditional classroom instruction by itself. Twenty-two students 
were randomly selected to use the OLI-Statistics course in hybrid mode, while 42 students were assigned 
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to the control group. The control group received traditional classroom instruction. Participants for both 
groups were selected from a pool of volunteers. The learning outcomes measured by the in-class exams 
scores of the OLI-Statistics group were compared with those of the instructor-led groups. Findings 
revealed that the OLI group completed the course in about half the time of the control group and achieved 
similar learning outcomes. 

Bowen, Chingos, Lack, and Nygren (2014) examined the learning outcomes of a hybrid machine-guided 
online interactive learning termed Interactive Learning Online (ILO). The study employed a randomized 
trial study design with two independent groups of public university students in six campuses in the United 
States. In essence, this study was a replication and extension of the Lovett et al. (2008) study. The first 
group was assigned to the traditional manual face-to-face teaching and learning while the other group 
received machine-guided instruction online. The researchers analyzed the learning outcomes of the two 
groups in the Carnegie Mellon University (CMU)-developed Statistics course. The course includes textual 
instructions and explanation, practice problems, and worked examples. They measured the groups’ 
learning outcomes on a standardized statistical literacy metric including pass rates and final exam scores. 
Findings revealed similar learning outcomes from the two groups. A speculative cost simulation analysis 
conducted by the authors, however, revealed significant cost savings with the hybrid machine-guided 
instruction learning model. 

Allen, Guzman-Alvarez, Molinaro, and Larsen (2015) assessed the impact and efficacy of the open-access 
ChemWiki Textbook in a general chemistry class at the University of California in a pilot study during the 
2014 Spring semester. The ChemWiki Textbook Project was planned to promote the use of open online 
textbooks as an alternative to the traditional hardcopy textbooks. The study employed an experimental 
design with an experimental group using the ChemWiki Textook in one chemistry class and a control 
group using a traditional chemistry text in the other chemistry class. The researchers analyzed the 
performance of the two groups, controlling for possible instructor bias by using the same set of instructors 
to teach both classes. The researchers examined grades obtained during in-class assessments, pre/post 
exam scores, and student attitude and self-reported study habits as measures of the students' learning 
outcomes. Although there was a difference in the students’ attitudes and beliefs between the two groups, 
the difference was not statistically significant. Likewise, no significant difference was found in the 
students’ learning outcomes or study habits between the two groups. 

The efficacy studies performed to date indicate that students who use OER do about as well or 
better as students who enroll in courses which use traditional materials. However, there are still 
limitations to the work that has been done to date. A particular weakness of more than half of the 
OER efficacy studies performed to date is that they make no effort to control differences in 
student characteristics or success metrics. 

The present study seeks to address this gap in the literature by examining the academic 
achievement (grades) of students enrolled in an introductory online history course which utilized 
OER compared to students enrolled in the same introductory online history course which used 
traditional textbooks. Specifically, the study attempted to answer this research question: What effect 
does enrollment in an OER course have on student achievement in that course when controlling for prior 
academic achievement? 
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Method 

The data for this study was collected from the student information system at Northern Virginia 
Community College (NOVA), a large, multi-campus urban community college located near Washington, 
D.C. The sample includes 146 students. These students were enrolled in an online 8-week section of an 
introductory history class in the Fall of 2013 or in the Spring of 2014. In Fall 2013, there were two sections 
of the course which utilized OER and two sections which did not. In Spring 2014, there were three 
sections of the course which utilized OER and three which did not. Of the students enrolled across all 
sections of the courses, 64% were female and 36% were male. Since the study utilized student 
information after the fact, there was no way to control for selection bias. 

The student information system allowed identification of the study’s predictor variables: student 
cumulative previous GPA prior to taking the online introductory history course (prior academic 
achievement) and identification of the course as either an OER version or a non-OER version. A 
correlation analysis was computed to assess the relationship between prior academic achievement and 
student grade in the course (student achievement). The correlation analysis also assessed the relationship 
between enrollment in an OER version of the introductory history course and student achievement. 

In addition, a multiple linear regression was calculated to predict student grade in the introductory 
history course (student achievement) based on OER and prior academic achievement. Dummy variables 
were created for course section identification with OER being the baseline group for comparison to non- 
OER sections of the introductory history course. Also, students’ final course grades were converted from 
letters to numbers, i.e. A=4, B=3, C=2, D=i, and F=o. For this study, W grades were converted to o. 


Results 

There was a moderate positive correlation between prior academic achievement and student achievement 
(r = .30, p < .001). There was an even stronger positive moderate correlation between OER versus Non- 
OER and student achievement (r = .41, p < .001). These results show that there is a moderately positive 
relationship between taking an OER course and academic achievement. 

A regression of student grade in the introductory histoiy course on student GPA prior to taking the online 
introductory history course and identification of the course as either an OER version or a non-OER 
version explained 22% of the variance in student Number Grade in the introductory histoiy course F (2, 
143) = 20.30, MSE = 1.67, p < .001. 

Student cumulative previous GPA was a significant predictor of student number grade in the introductory 
history course ((3 = .34, p < .01), which indicates that a .34 increase in previous student GPA led to a one 
standard deviation increase in student grade in the introductory history course after controlling for OER 
vs. non-OER. Enrollment in an OER course was also a significant predictor of student grade achievement 
(b = - 37 , P <.001). 
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Discussion 

The results indicate that there is a positive relationship between the use of OER and student academic 
achievement in an online history course. Because of the lack of demographic variables, we cannot 
definitively conclude that OER leads to increased student achievement. However, by controlling for prior 
college achievement, we show there may be evidence to support this claim. Students enrolled in the OER 
sections of the introductory history course performed as well or better than students enrolled in sections 
that used commercial textbooks. Usually, comparative studies examine the mode of delivery or 
instructional design as possible factors influencing student learning outcomes (Russell, 2015). One factor 
that this study was able to control for was the instructional design of the course. This study examined final 
course grades of students enrolled in 8-week sections of online courses with an identical instructional 
design. 

Other factors were not controlled in this study. First, the study did not control for teacher effect. Even in 
online courses created from a master template, teacher effect can impact student learning. How often an 
instructor posts announcements, responds to class discussion forums, answers student email inquiries, 
and grades student work can have an impact on student success in a course. The study also did not control 
for student characteristics besides previous GPA, which may also have a significant impact on academic 
achievement. It would be useful to examine the interaction effect of these variables in future studies that 
examine the impact of OER on student academic achievement. 


Conclusions and Future Research 

While the results of the study may not have implications for OER efficacy per se, the study is important 
for showcasing the ways that researchers in community colleges can begin to use data available at their 
institutions to study the effect of OER on student learning. While a more robust study, which includes a 
greater sample size and controls for confounding variables is needed, certainly this study provides the first 
step in developing an experimental research design that does have implications for future OER research. 

Further research is needed within other contexts to determine which elements of OER seem to have the 
greatest impact on student learning outcomes. Also, since this study was limited to one online course over 
the duration of one academic year, similar experiments, which control for various potential confounding 
variables must be undertaken in larger contexts to substantiate or repudiate the findings of this study. 
Finally, future studies may also examine the ways in which OER interact with particular subject areas and 
what text characteristics make some OER better than others. 

This study may be helpful for community college leaders who want to begin to use data more effectively or 
who seek to take on more of researcher/practitioner role at their institutions. By presenting a simple 
research model on which to build, this study can motivate community college leaders to conduct research 
of their own using data available from their own institutions to make decisions about OER use, adoption, 
and policy. 
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